[Anima] Add img2img pipeline blocks by PreethamNoelP · Pull Request #13929 · huggingface/diffusers

PreethamNoelP · 2026-06-12T15:29:51Z

What does this PR do?

This PR adds image-to-image (img2img) support to the Anima modular pipeline, as requested in #13903. The implementation follows the z_image modular pattern: img2img is a workflow inside the unified AnimaAutoBlocks, selected automatically when an image (or pre-encoded image_latents) is provided.

Design

New blocks

AnimaImg2ImgVaeEncoderStep — preprocesses the input image and encodes it with the VAE, returning image_latents only. Encoding is independent of strength / num_inference_steps / num_images_per_prompt, so latents encoded once can be reused across settings.
AnimaImageInputStep — input processing step (analogous to AnimaTextInputStep) that expands image_latents to the final denoising batch (batch_size * num_images_per_prompt) and derives height/width from the latents when not provided.
AnimaImg2ImgSetTimestepsStep — computes the full timestep schedule, then slices it by strength via get_timesteps(), which also sets the scheduler's begin index.
AnimaImg2ImgPrepareLatentsStep — generates noise and mixes it with the image latents via scheduler.scale_noise() at the first sliced timestep, and creates the Cosmos padding mask.
AnimaImg2ImgCoreDenoiseStep — flat SequentialPipelineBlocks: text_conditioning → input → image_input → set_timesteps → prepare_latents → denoise, mirroring ZImageImage2ImageCoreDenoiseStep.
AnimaAutoVaeImageEncoderStep — AutoPipelineBlocks triggered by image; skipped for text2image.
AnimaAutoDenoiseStep — AutoPipelineBlocks with block_trigger_inputs = ["image_latents", None], selecting img2img or text2image core denoise.

Unified pipeline

AnimaAutoBlocks = text_encoder → vae_encoder (auto) → denoise (auto) → decode, with a _workflow_map covering both text2image and img2img. No separate img2img pipeline class.

Usage

import torch
from diffusers import AnimaModularPipeline
from diffusers.modular_pipelines.anima import AnimaAutoBlocks

pipe = AnimaModularPipeline(
    blocks=AnimaAutoBlocks(),
    pretrained_model_name_or_path="circlestone-labs/Anima-Base-v1.0-Diffusers",
)
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = pipe(
    prompt="masterpiece, best quality, mountain landscape at golden hour, snow-capped peaks, pine forest, dramatic clouds, cinematic, detailed",
    image=input_image,
    strength=0.75,
    num_inference_steps=28,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

Test results

pytest tests/modular_pipelines/anima/ -q
28 passed, 10 skipped

Fixes #13903

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Link: [Anima] Add img2img capability #13903
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

Who can review?

@asomoza @yiyixuxu

…anima-img2img

asomoza · 2026-06-15T12:16:21Z

thanks @PreethamNoelP, can we start with updating the PR with a code snippet to test it and the images to see that it works before doing the review.

PreethamNoelP · 2026-06-15T16:43:07Z

Hi @asomoza, here is the code snippet and output images.

Usage:

import torch
from diffusers import AnimaModularPipeline
from diffusers.modular_pipelines.anima import AnimaImg2ImgAutoBlocks

pipe = AnimaModularPipeline(
    blocks=AnimaImg2ImgAutoBlocks(),
    pretrained_model_name_or_path="circlestone-labs/Anima-Base-v1.0-Diffusers",
)
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = pipe(
    prompt="masterpiece, best quality, mountain landscape at golden hour, snow-capped peaks, pine forest, dramatic clouds, cinematic, detailed",
    image=input_image,
    strength=0.75,
    num_inference_steps=28,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

Outputs:

asomoza · 2026-06-17T18:37:20Z

+# Copied from diffusers.modular_pipelines.anima.before_denoise.AnimaSetTimestepsStep
+class AnimaImg2ImgSetTimestepsStep(ModularPipelineBlocks):


you used the copied from here but you changed the function no? You can catch these errors and the code quality by following the contributing doc

Removed the # Copied from tag above AnimaImg2ImgSetTimestepsStep.
Since the class intentionally skips the set_begin_index(0) call (that's handled later by the VAE encoder step), it was never truly identical to the source, so the tag shouldn't have been there.
Please let me know if any further changes are required.

yiyixuxu · 2026-06-17T18:47:14Z

+
+
+# auto_docstring
+class AnimaImg2ImgAutoBlocks(SequentialPipelineBlocks):


we do not need a AnimaImg2ImgAutoBlocks. We add a workflow into the AnimaAutoBlocks

see example https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/z_image/modular_blocks_z_image.py#L325

Removed AnimaImg2ImgAutoBlocks entirely and folded img2img into AnimaAutoBlocks following the z_image pattern.
I added AnimaAutoDenoiseStep (an AutoPipelineBlocks that picks between img2img and txt2img based on whether image is provided) and AnimaImg2ImgDenoiseStep (a SequentialPipelineBlocks that wraps the img2img-specific set_timesteps + denoise steps).
AnimaAutoBlocks now has a _workflow_map covering both workflows, so users only need one pipeline - passing image= automatically triggers img2img.
Please let me know if any further changes are required.

…nto AnimaAutoBlocks - Remove incorrect `# Copied from` comment above AnimaImg2ImgSetTimestepsStep - Delete AnimaImg2ImgAutoBlocks; introduce AnimaAutoDenoiseStep (AutoPipelineBlocks) and AnimaImg2ImgDenoiseStep (SequentialPipelineBlocks) so img2img lives as a workflow inside AnimaAutoBlocks, following the z_image pattern - Update __init__.py, dummy_objects, and docs to remove AnimaImg2ImgAutoBlocks - Update img2img test to use AnimaAutoBlocks with updated workflow block paths

asomoza · 2026-06-26T19:05:44Z

@PreethamNoelP thanks, sorry I missed the update, I checked it and it looks good to me, I tested it and the outputs seems coherent too.

@yiyixuxu can you review it one last time to see if I missed something?

asomoza · 2026-06-26T19:08:49Z

@bot /style

github-actions · 2026-06-26T19:10:14Z

Style bot fixed some files and pushed the changes.

HuggingFaceDocBuilderDev · 2026-06-26T19:21:09Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ghunkins · 2026-07-02T14:55:53Z

Any updates on this by chance? Would love to see this in main! cc @yiyixuxu

yiyixuxu · 2026-07-02T19:04:59Z

+        )
+
+    @property
+    def inputs(self) -> list[InputParam]:


the encoder step should just take an image (height/width too) and return image_latents,

i.e. if you have already encoded the image once, you should not need to encode again just to use a different strength/num_images_per_prompt or num_inference_steps etc

see https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/z_image/encoders.py#L272

Hi @yiyixuxu, thank you for the suggestion. AnimaImg2ImgVaeEncoderStep has been refactored to only handle image encoding, returning image_latents. The timestep slicing and noise mixing logic has been extracted into a new AnimaImg2ImgPrepareLatentsStep, following the same separation as ZImageVaeImageEncoderStep and ZImagePrepareLatentswithImageStep.

yiyixuxu · 2026-07-02T19:09:16Z

+
+    block_classes = [AnimaImg2ImgDenoiseStep, AnimaCoreDenoiseStep]
+    block_names = ["img2img", "text2image"]
+    block_trigger_inputs = ["image", None]


Suggested change

block_trigger_inputs = ["image", None]

block_trigger_inputs = ["image_latents", None]

Hi @yiyixuxu, thank you for the suggestion. AnimaAutoDenoiseStep has been updated to use block_trigger_inputs = ["image_latents", None]. Additionally, AnimaAutoVaeImageEncoderStep (triggered by "image") has been added as a sequential step in AnimaAutoBlocks, ensuring image_latents is placed in state before the denoise step selects the appropriate block - consistent with the ZImageAutoBlocks structure.

…ImageEncoderStep - AnimaImg2ImgVaeEncoderStep now only encodes image → image_latents - AnimaImg2ImgPrepareLatentsStep handles timestep slicing + noise mixing - AnimaAutoVaeImageEncoderStep (AutoPipelineBlocks, trigger: image) added to AnimaAutoBlocks - AnimaAutoDenoiseStep trigger changed from image to image_latents - Follows z_image pattern (ZImageVaeImageEncoderStep + ZImagePrepareLatentswithImageStep)

… refactoring

yiyixuxu · 2026-07-09T00:28:22Z

+        block_state.timesteps, block_state.num_inference_steps = get_timesteps(
+            components.scheduler, block_state.num_inference_steps, block_state.strength
+        )


can we move this into AnimaImg2ImgSetTimestepsStep? makes more sense there

Hi @yiyixuxu, thank you for the suggestion.
The get_timesteps() call has been moved into AnimaImg2ImgSetTimestepsStep, which now computes the full schedule and slices it by strength in a single step. AnimaImg2ImgPrepareLatentsStep no longer touches the timestep schedule.

yiyixuxu · 2026-07-09T00:31:29Z

+            components.scheduler, block_state.num_inference_steps, block_state.strength
+        )
+
+        total_batch = block_state.batch_size * block_state.num_images_per_prompt


in modular, we should expand the image_latent in the input step , you can add an image_input step similar to https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/anima/before_denoise.py#L196

Hi @yiyixuxu, thank you for the suggestion.
A new AnimaImageInputStep has been added following the same pattern as AnimaTextInputStep - it expands image_latents to batch_size * num_images_per_prompt via repeat_tensor_to_batch_size and derives height/width from the latents when not provided. AnimaImg2ImgPrepareLatentsStep now only handles noise generation and scale_noise().

yiyixuxu · 2026-07-09T00:34:33Z

+        return [OutputParam.template("latents")]
+
+
+# auto_docstring


why do we need this? can you just add the set_timesteps as part of core_denoise?

Hi @yiyixuxu, you're right - the wrapper was left over from an earlier structure and is no longer needed. AnimaImg2ImgDenoiseStep has been removed, and AnimaImg2ImgSetTimestepsStep is now part of AnimaImg2ImgCoreDenoiseStep directly, making it a flat sequence (text_conditioning → input → image_input → set_timesteps → prepare_latents → denoise) consistent with ZImageImage2ImageCoreDenoiseStep.

… image input step, flatten img2img core denoise

yiyixuxu · 2026-07-16T04:40:21Z

+
+
+# auto_docstring
+class AnimaAutoDenoiseStep(AutoPipelineBlocks):


Suggested change

class AnimaAutoDenoiseStep(AutoPipelineBlocks):

class AnimaAutoCoreDenoiseStep(AutoPipelineBlocks):

Hi @yiyixuxu, thank you for the suggestion.
AnimaAutoDenoiseStep has been renamed to AnimaAutoCoreDenoiseStep, consistent with the naming used across the other modular pipelines.

yiyixuxu · 2026-07-16T04:43:09Z

+enable_full_determinism()
+
+
+ANIMA_IMG2IMG_WORKFLOWS = {


can you put into same file https://github.com/huggingface/diffusers/blob/main/tests/modular_pipelines/anima/test_modular_pipeline_anima.py

Hi @yiyixuxu, done - the img2img test class has been moved into test_modular_pipeline_anima.py and the separate test file has been removed.

…enoiseStep, merge img2img tests into main test file

yiyixuxu

thanks

PreethamNoelP added 2 commits June 12, 2026 20:20

[Anima] Add img2img pipeline blocks

585deff

Merge branch 'main' of https://github.com/huggingface/diffusers into …

56664b7

…anima-img2img

github-actions Bot added documentation Improvements or additions to documentation tests modular-pipelines utils fixes-issue size/L PR with diff > 200 LOC labels Jun 12, 2026

asomoza reviewed Jun 17, 2026

View reviewed changes

yiyixuxu reviewed Jun 17, 2026

View reviewed changes

github-actions Bot removed documentation Improvements or additions to documentation utils labels Jun 19, 2026

Merge branch 'main' into anima-img2img

adb1808

asomoza approved these changes Jun 26, 2026

View reviewed changes

Apply style fixes

9e05f28

yiyixuxu reviewed Jul 2, 2026

View reviewed changes

PreethamNoelP added 3 commits July 4, 2026 08:54

Merge remote-tracking branch 'upstream/main' into anima-img2img

7d18770

Merge origin/anima-img2img: resolve docstring conflicts, keep round-2…

d5712bc

… refactoring

yiyixuxu reviewed Jul 9, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into anima-img2img

cec4f26

PreethamNoelP added 2 commits July 10, 2026 18:06

[Anima] Address review: move strength slicing into set_timesteps, add…

287a89c

… image input step, flatten img2img core denoise

[Anima] Fix doc-builder style: rewrap docstrings to 119-char width

2470785

PreethamNoelP requested a review from yiyixuxu July 16, 2026 03:03

yiyixuxu reviewed Jul 16, 2026

View reviewed changes

PreethamNoelP added 2 commits July 16, 2026 16:11

[Anima] Address review: rename AnimaAutoDenoiseStep to AnimaAutoCoreD…

f573cd8

…enoiseStep, merge img2img tests into main test file

Merge remote-tracking branch 'upstream/main' into anima-img2img

c23411d

PreethamNoelP requested a review from yiyixuxu July 16, 2026 11:09

yiyixuxu approved these changes Jul 16, 2026

View reviewed changes

yiyixuxu merged commit e11810a into huggingface:main Jul 16, 2026
18 of 19 checks passed

		# Copied from diffusers.modular_pipelines.anima.before_denoise.AnimaSetTimestepsStep
		class AnimaImg2ImgSetTimestepsStep(ModularPipelineBlocks):



		# auto_docstring
		class AnimaImg2ImgAutoBlocks(SequentialPipelineBlocks):

	block_trigger_inputs = ["image", None]
	block_trigger_inputs = ["image_latents", None]



		# auto_docstring
		class AnimaAutoDenoiseStep(AutoPipelineBlocks):

	class AnimaAutoDenoiseStep(AutoPipelineBlocks):
	class AnimaAutoCoreDenoiseStep(AutoPipelineBlocks):

		enable_full_determinism()


		ANIMA_IMG2IMG_WORKFLOWS = {

Uh oh!

Conversation

PreethamNoelP commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Design

Usage

Test results

Before submitting

Who can review?

Uh oh!

asomoza commented Jun 15, 2026

Uh oh!

PreethamNoelP commented Jun 15, 2026

Uh oh!

asomoza Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PreethamNoelP Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asomoza commented Jun 26, 2026

Uh oh!

asomoza commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jun 26, 2026

Uh oh!

ghunkins commented Jul 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PreethamNoelP commented Jun 12, 2026 •

edited

Loading

asomoza Jun 17, 2026 •

edited

Loading

PreethamNoelP Jun 19, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading